Yazilim Test Maliyet Fonksiyonlarinin Otomatik Olarak Kesfedilmesi

The testing of highly configurable systems almost always involves sampling enormous configuration spaces and testing representative instances of a system’s behavior. This sampling can be done by computing a combinatorial object, called a t-way covering array (CA). The covering arrays assume that the cost of configuring the system under test is the same for all configurations, however this is not a practical assumption. To compute cost-aware covering arrays, the cost needs to be determined beforehand. Therefore, estimating the cost of a quality assurance (QA) task across a configuration space is of great importance, as the estimates can be used for planning the QA process as well as for taking cost-aware samples. However, manually creating cost models is cumbersome and error-prone, thus impractical. Therefore we have been developing automated approaches for cost model discovery in configuration spaces. In our previous work, we have computed generalized linear regression models from the data set which contains the measured costs of all configurations in a covering array for a given QA task. In this paper, we have developed another approach using Design of Experiments Theory (DoE) for automatically discovering the cost function and compared it with our previous approach based on linear regression models. Given a configuration space, a QA task of interest, and a cost of the QA task, the proposed approach first identifies important effects, i.e., combinations of option settings that affect the cost most, by using screening designs from the DoE theory, and then uses the important effects identified to fit a cost model to the observations. To evaluate the proposed approach, we used 3 different QA tasks (1To build the system under test 2To run a single test case 3To run a whole test suite) on 2 different real software systems (Apache web server and MySQL database server). These models computed by both the generalized linear regression and screening designs have been evaluated by the coefficient of determination metric known as R-squared in statistics and the results have been successful with an average measure of 0.92 and 0.99.